Electronic apparatus having a sensing unit to input a user command and a method thereof

ABSTRACT

An electronic apparatus includes a housing, a display unit installed in the housing and having a display area to display an image or a menu icon, a sensing unit installed in the housing adjacent to the display unit, having a virtual area to sense an object which is disposed over the display area of the display unit, and having a first camera to photograph and display the virtual area in a photographing mode and a second camera to extract the object from the virtual area and display a sensed image of the extracted object in a sensing mode, and a controller to analyze status of the object according to the sensed image of the object, and to determine the analyzed status of the object as a user input in the sensing mode.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation Application of U.S. Non-Provisionalapplication Ser. No. 15/440,033, filed on Feb. 23, 2017, now U.S. Pat.No. 10,013,115, which is a Continuation application Ser. No. 15/246,302,filed on Aug. 24, 2016, now U.S. Pat. No. 9,864,511, which is aContinuation application Ser. No. 14/583,537, filed on Dec. 26, 2014,now U.S. Pat. No. 9,454,235, in the United States Patent and TrademarkOffice, the contents of which are incorporated herein in their entiretyby reference.

BACKGROUND OF THE INVENTIVE CONCEPT

1. Field of the Inventive Concept

The present inventive concept relates to an electronic apparatus andmethod of inputting a user command through a sensing unit thereof.

2. Description of the Related Art

A conventional electronic mobile device, for example, a cellular phone,includes a camera and a display unit having a display area to display animage photographed by the camera. However, a center line of the cameradoes not correspond to a center line of the display area. Therefore,when an object is disposed over a position of the center line of thedisplay area, an image of the object is displayed on a positiondifferent from the center line of the display area of the display unit.

When a user intends to place an object on a position to be disposed overthe display area to correspond to a position of the display area in avertical direction relationship between the position of the object andthe position of the display area, an image of the object displayed onthe display area is deviated from the intended position.

When a user intends to move an object between positions to be disposedover the display area to correspond to positions of the display area ina vertical direction relationship between the positions of the objectand the positions of the display area, an image of the object displayedon the display area is deviated from the intended positions.

Since a position of an object is sensed to be different from a positionof a display area, it is difficult to sense a position of an objectsensed by a camera to correspond to a position of a point of a displayarea.

SUMMARY OF THE INVENTIVE CONCEPT

The present inventive concept provides an electronic apparatus to inputa user command through a sensing unit having a virtual area overlappinga display area in a vertical direction of the display area in anelectronic apparatus.

The present inventive concept provides a method of inputting a usercommand through a sensing unit having a virtual area overlapping adisplay area in a vertical direction of the display area in anelectronic apparatus.

The present inventive concept provides a computer readable medium tocontain computer readable codes as a program to execute a method ofinputting a user command through a sensing unit having a virtual areaoverlapping a display area in a vertical direction of the display areain an electronic apparatus.

Additional features and utilities of the present inventive concept willbe set forth in part in the description, which follows and, in part,will be obvious from the description, or may be learned by practice ofthe present inventive concept.

The foregoing and/or other features and utilities of the presentinventive concept may be achieved by providing a mobile device includinga housing having a front side and a rear side, an input and outputinstalled in the housing, and comprising a panel on the front side ofthe housing to receive a user input and to display a photographing imageor a menu, a sensing unit installed on the front side of the housing,and having a first camera and a second camera each having a virtual areacorresponding to a display area of the panel of the input and outputunit, and a controller installed in the housing, and configured tocontrol at least one of the first camera and the second camera, whereinthe controller is configured in a photographing mode to control thefirst camera to photograph the virtual area to generate a firstphotographing image, and to control the display unit to display thefirst photographing image in the display area of the panel of the inputand output unit, and wherein the controller is configured in a sensingmode to control the second camera to photograph an object of the virtualarea disposed to correspond to the display area of the panel of theinput and output unit, and is configured to perform a function of themobile device according to the menu and a state of the object of thevirtual area.

The mobile device may further include a memory to store information, andthe controller may compare the information and the state of the objectand may perform the function of the mobile device according to the menuand the comparison.

The controller may analyze the photographed object of the virtual areato sense the state of the object, and to perform the function of themobile device according to the menu and the state of the object of thevirtual area.

The controller may modify the photographed object of the virtual areawith respect to the display area to sense the state of the object, andmay perform the function of the mobile device according to the menu andthe state of the object of the virtual area.

The controller may determine the state of the object of the virtual areaas a user input to the menu, and may perform a function of the mobiledevice according to the menu and the user input.

The controller may control the second camera to photograph the object ofthe virtual area when the menu is displayed in the display area.

The controller may control the second camera to generate a secondphotographing image corresponding to the virtual area, and may extractthe object from the second photographing image to sense the state of theobject.

The controller may display an image corresponding to the object withrespect to the menu displayed on the display area of the panel of theinput and output unit.

The controller may control the input and output unit to generate anoutput of the function of the mobile device. The input and output unitmay include an audio element, and the output may include an image outputto be displayed on the panel or an audio output to be generated from theaudio element.

The mobile device may further include an interface configured tocommunicate with an external device to transmit or receive data toperform a function of the mobile device. The controller may beconfigured to perform the function of the mobile device according to thedata from the external device, the menu, and the state of the object asthe user input to the menu.

The virtual area may include a plurality of sub-objects, one of whichhas a higher priority than others, and the controller may select the onesub-object as the object corresponding to the menu.

The foregoing and/or other features and utilities of the presentinventive concept may also be achieved by providing a method of a mobiledevice, the method including providing a housing having a front side anda rear side, an input and output installed in the housing and comprisinga panel on the front side of the housing to receive a user input and todisplay a photographing image or a menu, and a sensing unit installed onthe front side of the housing, and having a first camera and a secondcamera each having a virtual area corresponding to a display area of thepanel of the input and output unit, using a controller processorinstalled in the housing, controlling the first camera in aphotographing mode to photograph the virtual area to generate a firstphotographing image, and to control the display unit to display thefirst photographing image in the display area of the panel of the inputand output unit, and using the controller processor, controlling thesecond camera in a sensing mode to photograph an object of the virtualarea disposed to correspond to the display area of the panel of theinput and output unit, and performing a function of the mobile deviceaccording to the menu and a state of the object of the virtual area.

The mobile device may further include a memory to store information, andthe method may further include comparing the information and the stateof the object and performing the function of the mobile device accordingto the menu and the comparison.

The method may further include analyzing the photographed object of thevirtual area to sense the state of the object, and performing thefunction of the mobile device according to the menu and the state of theobject of the virtual area.

The method may further include modifying the photographed object of thevirtual area with respect to the display area to sense the state of theobject, and performing the function of the mobile device according tothe menu and the state of the object of the virtual area.

The method may further include determining the state of the object ofthe virtual area as a user input to the menu, and performing thefunction of the mobile device according to the menu and the user input.

The second camera may be controlled to photograph the object of thevirtual area when the menu is displayed in the display area.

The method may further include generating an output of the functionthrough the panel or an audio element of the input and output unit.

The virtual area may include a plurality of sub-objects, one of whichhas a higher priority than others, and the method may further includeselecting the one sub-object as the object corresponding to the menu.

The foregoing and/or other features and utilities of the presentinventive concept may also be achieved by providing a non-transitorycomputer-readable medium containing computer-readable codes as a programto perform the method described above or hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other features and utilities of the present generalinventive concept will become apparent and more readily appreciated fromthe following description of the exemplary embodiments, taken inconjunction with the accompanying drawings of which:

FIG. 1 is a diagram illustrating an electronic apparatus according to anembodiment of the present inventive concept;

FIG. 2 is a block diagram illustrating an electronic apparatus accordingto an embodiment of the present inventive concept

FIGS. 3 and 4 are views illustrating a sensing unit and a display screenin an electronic apparatus according to an embodiment of the presentgeneral inventive concept;

FIGS. 5 and 6 are views illustrating an object sensed by a sensing unitof an electronic apparatus according to an embodiment of the presentinventive concept;

FIGS. 7 and 8 are views illustrating a point of a virtual area tocorrespond to a display point of a display area in an electronicapparatus according to an embodiment of the present inventive concept;

FIGS. 9 and 10 are views illustrating a status of an objectcorresponding to a virtual point with respect to a virtual area and adisplay area in an electronic apparatus according to an embodiment ofthe present inventive concept;

FIGS. 11 and 12 are views illustrating objects sensed by a sensing unitin an electronic apparatus according to an embodiment of the presentinventive concept;

FIG. 13 is a view illustrating a location of a sensing unit in anelectronic apparatus according to an embodiment of the present inventiveconcept;

FIGS. 14A, 14B, 15A, and 15B are views illustrating a virtual area of asensing unit and a display area of a display unit according to anembodiment of the present inventive concept;

FIG. 16 is a view illustrating a location of a sensing unit in anelectronic apparatus according to an embodiment of the present inventiveconcept;

FIG. 17 is a view illustrating a virtual area of a sensing unit and adisplay area of a display unit according to an embodiment of the presentinventive concept;

FIGS. 18A and 18B are views illustrating a virtual point displayed on adisplay area of a display unit in an electronic apparatus according toan embodiment of the present inventive concept;

FIGS. 19 and 20 are flowcharts illustrating a method of inputting a usercommand using a sensing unit in an electronic apparatus according to anembodiment of the present inventive concept;

FIG. 21 is a view illustrating a location of a sensing unit in anelectronic apparatus according to an embodiment of the present inventiveconcept; and

FIGS. 22 and 23 are views illustrating an electronic apparatus having asensing unit installed thereon according to an embodiment of the presentgeneral inventive concept.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

Reference will now be made in detail to exemplary embodiments of thepresent general inventive concept, examples of which are illustrated inthe accompanying drawings, wherein like reference numerals refer to thelike elements throughout. The exemplary embodiments are described belowin order to explain the present general inventive concept whilereferring to the figures.

FIG. 1 is a diagram illustrating an electronic apparatus 100 accordingto an embodiment of the present inventive concept. The electronicapparatus 100 may be a mobile device having one or more camera units tophotograph an object disposed in a front side and/or a back side of themobile device. The electronic apparatus 100 includes a housing 101, adisplay unit 102 disposed on the housing 101 to display an image on adisplay area DA having a display center line DCL, and a sensing unit 103having a sensing element to sense an object disposed around theelectronic apparatus 100.

The sensing unit 103 may include the one or more camera units having asensing center line SCL to generate an image corresponding to thephotographed object with respect to the display area DA. The sensingcenter line SCL and the display center line DCL are spaced apart fromeach other. Although the sensing center line SCL is illustrated in aleft-right direction of FIG. 1, the sensing center line SCL may bedisposed at a line perpendicular to a surface of the housing 101, forexample, the display area DA. Although the display center line DCL isillustrated in a left-right direction of FIG. 1, the display center lineDCL may be disposed at a line perpendicular to a surface of the housing101, for example, the display area DA.

As an example, when the sensing unit is a camera to photograph anobject, and the display area DA displays an image corresponding to thephotographed object, a center of the displayed image of the photographedobject may be not identical to a center of the photographed object dueto a distance between the sensing center line SAC and the display centerline DCL. When an object approaches toward a portion of the display areaDA in a direction perpendicular to a surface of the display area DA, theimage of the photographed object is not displayed to move toward theportion of the display area DA in a direction perpendicular to a surfaceof the display area DA. As another example, when an object moves towardor disposed at a center of the display area DA, the image of thephotographed object is not displayed at the center of the display areaDA but is displayed at another portion spaced apart from the center ofthe display area DA.

When a user is disposed in front of the electronic apparatus to face thedisplay area DA and a user finger or tool is disposed over or movestoward a point of the display area DA, a photographed image of the userfinger or tool is displayed on the display area DA differently as if theuser finger or tool is disposed over or moves toward another point ofthe display area DA. The photographed position (location) or movement ofthe user finger or tool may be referred to a virtual position (location)or movement of the object, and the position (location) or movement ofthe object with respect to the display area DA may be referred to anactual or physical position (location) or movement of the object. Thevirtual position or movement of the object in the photographed or sensedimage with respect to the display area DA may be different from theactual position or movement with respect to the display area DA.Therefore, the electronic apparatus may not determine or recognize thestatus of the object as a user input according to the virtual positionor movement in the photographed or sensed image but may generate anerror in determining or recognizing the virtual position of movement ofthe object as a user input with respect to the display area DA.Accordingly, the virtual position (location) or movement of thephotographed image of the object may be modified or corrected to theactual or physical position (location) or movement such that the statusof the object can be determined as a user input with respect to a pointof the display area DA. This modification or correction allows theelectronic apparatus to determine or recognize a user input from thephysical location or movement of the object with respect to the displayarea DA.

FIG. 2 is a block diagram illustrating an electronic apparatus 200according to an embodiment of the present inventive concept. Theelectronic apparatus 200 may be the electronic apparatus 100.

The electronic apparatus 200 includes a controller 210, a memory unit220, an interface 230, a sensing unit 203, and an input/output unit 204.The controller 210, the memory 220, the interface 230, the sensing unit203 and the input/output unit 204 may be included in a single housing.However, it is possible that at least one of the memory 220, theinterface 230, the sensing unit 203 and the input/output unit 204 may beconnected to the housing. The sensing unit 203 may correspond to thesensing unit 103 of FIG. 1, and the input/output unit 204 may correspondto the display unit 102 of FIG. 1. The memory unit 220 may store datausable to perform functions of the electronic apparatus 200. The memoryunit 220 may be a semiconductor memory device, a read-only memory (ROM),a random-access memory (RAM), a flash memory, a semiconductor chippackage, CD-ROMs, magnetic tapes, floppy disks, and optical data storagedevices. The input/output unit 204 may be a display device to display animage or a touch panel to receive a user input. The input/output unit204 may include an audio element to generate an audio signal alone ortogether with an image displayed on a display unit according to a userinput or data processed in the controller 210.

The sensing unit 203 may include one or more cameras to photograph anobject and/or a sensing element to sense an object disposed over adisplay area of the display device or touch panel. The photographed orsensed object may be displayed as an image on the display area DA.However, it is possible that the photographed or sensed object may notbe displayed as an image but may be usable as a user input (command ordata) to correspond to a portion (icon or menu) of the display area DA.It is also possible that the photographed or sensed object may not bedisplayed as an image of the photographed object but may be displayed asa symbol representing the photographed or sensed object with respect tothe display area DA. That is, when the sensing unit 203 is in a cameramode, the photographed object is displayed as an image on the displayarea DA. When the sensing unit 203 is in a sensing mode, the sensedobject is usable as a user input (or user data input) to correspond to aportion (icon or menu selection or a user command) of the display areaDA. It is also possible that in a combination mode, the photographedobject is displayed on the display area DA, the icon or menu isdisplayed on the display area DA, and the photographed (or sensed)object is used as a user input to control or perform a function of theicon (or menu) of the display area or to control the displayed image orto control the sensing unit 203. The camera mode, the sensing mode, andthe combination mode can be selected in a menu screen of the displayarea DA or can be selected according to a selection of a buttoncorresponding to the respective mode according to a user input.

The input/output unit 204 may be a display unit to display an image,and/or an input unit to input a user input (command or data). Theinput/output unit 204 may have a display area DA of a screen (panel) todisplay an image photographed by a camera, to display a menu icon, toreceive a user touch as a user input through the displayed menu icon,and to display an image sensed by the sensing element. The panel may bea touch panel. The touch panel is well known, detail descriptionsthereof are omitted. The input/output unit 204 may separately orindependently generate or receive the photographed image, the displayedmenu icon, the received user input, and the sensed image incorresponding modes according to a user design or preference. It ispossible that the input/output unit 204 may simultaneously generate orreceive the photographed image, the displayed menu icon, the receiveduser input, and the sensed image in corresponding modes according to auser design or preference. It is also possible that the input/outputunit 204 may generate a combination of the photographed image, thedisplayed menu icon, the received user input, and the sensed imageaccording to a user design or preference.

The controller 210 may generate a menu option to be displayed on thedisplay area DA. The menu option may include a first menu option toselect a camera mode in which a camera photographs an object andgenerates the photographed image to be displayed on the display area DA,a sensing mode in which a sensing unit senses an object disposed overthe display area DA and generates the sensed image corresponding to thesensed object to be displayed on the display area DA, and a combinationmode in which a camera is usable as a camera and a sensing element toperform both functions of the camera and the sensing element. Here, thephotographed image is an image photographed by a camera and the sensedimage is a symbol of the sensed object. Therefore, when one or moreobjects are photographed, the photographed objects are displayed on thedisplay area DA, and at least one of the photographed objects may besensed as the sensed image to be displayed as a symbol representing atleast a portion of the one object. The photographed image and the sensedimage may overlap on the display area DA. And also the menu icon and thesensed image may overlap on the display area DA. The sensed image may beuseable as a user input corresponding to a menu icon displayed on thedisplay area DA of the screen so that the controller 210 can determinethat a menu icon is selected according to a location of the sensed imageand can perform a function of the menu icon corresponding to the sensedimage.

The controller 210 may receive the photographed (or sensed) object fromthe sensing unit 203 and generate an image corresponding to thephotographed object to be displayed on the display area. Here, it ispossible that the controller 210 controls or modifies the image todisplay a portion of the photographed object which is disposed above thedisplay area DA in a direction, for example, in a directionperpendicular to the display area DA such that a center line of thedisplayed portion of the photographed object is identical to the centerDCL of the display area DA or such that an area of the display portionof the photographed object is identical to the display area DA. Thesensing unit 203 may perform at least a portion of the above describedfunction of the controller 210. According to status of the photographedor sensed object, the controller may receive a user input or command toperform a function including, but not limited to, a drag operation, aselecting operation, a data input operation, a displayed imagemodification operation, an communication method selecting operation, adata transmission or reception method selecting method, an externaldevice selecting operation, an access point selecting operation, and soon. The status of the object may be a stationary or a motion between twopositions in a virtual area VA of the sensing unit 203 with respect tothe display area DA of the input/output unit 204.

As stated above, due to different center lines of the display area DAand the sensing unit 203, a location or movement of the photographedobject with respect to the display area DA is displayed at a differentlocation or movement within the display area DA. When an object isdisposed at a position over a center of the display area DA in adirection perpendicular to the display area DA, the photographed objectis displayed at a different position from the center of the display areaDA. However, the photographed object disposed at a position over acenter of the display area DA in the perpendicular direction may bedisplayed at the center of the display area DA by analyzing andmodifying a photographed object image. The controller 210 may control,adjust, or modify a location or movement of an object photographed withrespect to the sensing unit 203 to correspond to a location and amovement with respect to display area DA. In this case, an objectdisposed at another portion which is not in an area corresponding to thedisplay area DA in the perpendicular direction may not be displayed inthe display area. An object disposed in an area corresponding to thedisplay area DA in the perpendicular direction may be displayed in thedisplay area. The sensing unit 203 may perform at least a portion of theabove described function of the controller 210.

The controller 210 may include a look-up table usable to analyze a firststate of the object in the photographed or sensed object in the virtualarea VA and to convert (correct or modify) the first state of the objectinto a second state of the object with respect to the display screen DAas the status of the object. The look-up table may include informationon coordinates of the virtual area VA and the display screen andinformation on a virtual location (or movement) of the object in thephotographed or sensed image in the virtual area VA and an actual orphysical location (or movement) of the object with respect to thedisplay area DA. When the object is disposed or moved in the virtualarea VA, the sensing unit 203 photographs the object to generate thephotographed or sensed image of the object with respect to thecoordinate of the virtual area VA. And then a location or movement ofthe photographed or sensed image of the object is analyzed and modifiedto a different location or movement with respect to the coordinate ofthe display screen. Since the coordinates of the virtual area VA and thedisplay area DA are same, the object is represented in the coordinate ofthe display area DA at a different location or movement from thelocation or movement in the virtual area VA. Although the coordinates ofthe virtual area VA and the display area DA are same, the virtuallocation or movement of the object photographed or sensed with respectto the virtual area VA may be different from the actual or physicallocation or movement of the object with respect to the display area DAaccording to a distance between the sensing center line SCL of thesensing unit 203 and the display center line SCL of the input/outputunit 204 or according to a distance from the sensing center line SCL tothe location of the object in the virtual area VA.

The interface 230 communicates with an external device 260 using a wiredor wireless communication method to transmit data to the external device260 and to receive data from the external device 260 according to a userinput. The controller 210 controls the above described units andelements of the electronic apparatus 200. The controller 210 may includeone or more semiconductor processors to perform functions of theelectronic apparatus 200.

FIG. 3 is a view illustrating a sensing area SA and a display area DA inan electronic apparatus according to an embodiment of the presentgeneral inventive concept. A camera (or sensing element) C1 as thesensing unit 203 of FIG. 2 photographs an object in the sensing area SAhaving a photographing angle “a” and a sensing center line SCL. Avirtual area VA is disposed in the sensing area SA to correspond to thedisplay area DA in a direction perpendicular to the display area DA. Thesensing center line SCL of the camera C1 may be parallel to a displaycenter line DCL of the display area DA. It is possible that sensingcenter line SCL of the camera C1 may not be parallel to a display centerline DCL of the display area DA to control or adjust a distance d1 whichis a distance between the display area DA and a boundary of a virtualplane VP or a boundary SAb of the sensing area SA. The distance D1 maybe adjustable when the camera C1 is adjustably installed in the housing101 of FIG. 1 to change the sensing center line SCL.

FIG. 4 is a view illustrating a sensing unit and a display screen in anelectronic apparatus according to an embodiment of the present generalinventive concept. The sensing unit 203 may have the camera C1 of FIG. 3to have a first sensing center line SCL1 and a first sensing area SA1,and a camera C2 to have a second sending center line C2 and a secondsensing area SA2. The virtual plane VP may be disposed in an overlaparea of the first sensing area SA1 and the second sensing area SA2. Thevirtual plane VP may be disposed between the first sensing center lineSCL1 and the second sensing center line SCL2. The virtual area VA isdisposed over the display area DA in a perpendicular direction withinthe overlap area. the virtual plane VP may be spaced apart from thedisplay area DA by a distance d2. The distance d2 may also be adjustableaccording to an adjustably installation of the camera C1 and the cameraC2.

FIGS. 5 and 6 are views illustrating an object sensed by a sensing unitof an electronic apparatus according to an embodiment of the presentinventive concept. One or more icons (or menus) Ma, Mb, . . . , Mn maybe displayed on the display area DA. the virtual area VA is disposedover the display area in a perpendicular direction. When an object, forexample, a user's finger Pob is disposed over the virtual area VA ormoves toward or away from the virtual area VA. The sensing unit 203senses the object disposed in the virtual area VA, and the controller210 may determine whether a location (or position) of the object isdisposed to correspond to one of the icons Ma, Mb, . . . , Mn of thedisplay area DA.

When the controller 210 determines that the object is disposed at alocation (or position) Vpa corresponding to the icon Ma of the displayarea DA, the controller 210 may determine that the icon Ma is selectedso that a corresponding function of the icon Ma can be performed. Thecontroller 210 may determine the presence or existence of the object inthe virtual area VA as a user input. The object can be represented by astationary object disposed at a position or a movable object movablebetween two positions. The controller 210 may control the input/outputunit 204 to display a character or sign Tpa within the correspondingicon Ma of the display area DA to represent that the icon Ma isselected. It is possible that the icon Ma is highlighted or emphasized.The controller 210 processes the function corresponding to the userinput and controls the input/output unit 204 to generate an output, forexample, displaying an image corresponding to the processed function,communicating with other device, or transmitting or receiving dataaccording to the processed function.

Since the display area DA includes a touch screen, the controller 210may receive a user input from a touch input though the input/output unit204 and/or a sensed object though the sensing unit 203. For example, atouch input on the icon MA of the display area DA and a sensed object inthe virtual area are determined as a user input to control theelectronic apparatus.

FIGS. 7 and 8 are views illustrating a point of a virtual area tocorrespond to a display point of a display area in an electronicapparatus according to an embodiment of the present inventive concept.An object, for example, a user finger Pob, moves toward or disposed at aposition VPa or VPb with respect to a display center line DCL of thedisplay area DA in the virtual area VA. The positions VPa and TPa ofFIG. 7 are opposite to the positions VPb and TPb of FIG. 8 with respectto the display center line DCL of the display area DA. All of thepositions VPa, VPb, TPa, TPb are disposed in a side of the sensingcenter line SCL of the sensing unit 203.

FIGS. 9 and 10 are views illustrating a status of an objectcorresponding to a virtual point with respect to a virtual area and adisplay area in an electronic apparatus according to an embodiment ofthe present inventive concept. When an object moves between twopositions with respect to a position VPa in a first direction of FIG. 9or in a second direction of FIG. 10, the controller 210 determines thepresence, existence, or movement of the object as a user input to selectthe icon Ma, and then perform the corresponding function. Although FIGS.9 and 10 illustrate two directions, the present general inventiveconcept is not limited thereto. The controller 210 may detect themovement of the object in a direction other than the two directions ofFIGS. 9 and 10.

FIGS. 11 and 12 are views illustrating objects sensed by a sensing unitin an electronic apparatus according to an embodiment of the presentinventive concept. When the sensing unit 203 senses two or more objects,the controller 210 may provide a priority to at least one of objects todetermine a user input. For example, when there are two objects in thevirtual area VA, the controller 210 may determine that a user finger Pobhas a higher priority than a user face as a user input. It is alsopossible that a distal end T of the user finger Pob can have a higherpriority than other portions of the user finger Pob as a user input. Theuser input can be determined according to the presence, existence, andmovement of the distal end T in the virtual area.

FIG. 13 is a view illustrating a location of a sensor of a sensing unitin an electronic apparatus according to an embodiment of the presentinventive concept. The sensing unit may be a camera having a lens unit,and the lens unit may have a photographing angle to define a sensingarea SA and a sensing center line SCL disposed at a center of thesensing area SA. The sensing area SA may have two sensing areas SAa andSAb which are disposed opposite to each other with respect to thesensing center area SCL. The lens unit may have a sensor which may notdisposed at a center area but disposed at a side Sa or Sb with respectto the sensing center line SCL to sense one of the two sensing areas SAaand SAb. In this case, one of the two sensing areas SAa and SAb may notbe sensed by the sensor disposed at the side.

FIGS. 14A, 14B, 15A, and 15B are views illustrating a virtual area of asensing unit and a display area of a display unit according to anembodiment of the present inventive concept. The virtual area VA todetect an object as a user input is disposed at one side of the sensingarea SA, for example, the virtual plane VP, with respect to the sensingcenter line SCL. The sensing area SA and the display area DA areillustrated in a vertical direction in FIGS. 14A and 14B and in ahorizontal direction in FIGS. 15A and 15B, according to a location ofthe sensing unit with respect to the display area DA.

FIG. 16 is a view illustrating a location of a sensing unit in anelectronic apparatus according to an embodiment of the present inventiveconcept. The sensing unit may be disposed at a center of the displayarea DA or at a position on the display center line DCL and/or thesensing center line SCL. The lens unit may be disposed at a bottom DPbof a display panel (touch panel) DP. The display panel DP may be aliquid crystal device (LCD) having a screen corresponding to the displayarea DA, a circuit element to form an image to be displayed on thescreen, and a light element to generate light toward the circuitelement. The sensing unit may be disposed at a center of the LCD. Inthis case, the LCD may provide a transparent condition to a portioncorresponding to the sensing unit such that the sensing unit senses theobject through the LCD. The light element may have a portioncorresponding to the sensing unit to prevent interference with thesensing operation of the sensing unit. The display panel DP may beformed with an LED unit having a number of LEDs. In this case, the LEDunit may have a portion without an LED to prevent interference with theoperation of the sensing unit.

FIG. 17 is a view illustrating a virtual area of a sensing unit and adisplay area of a display unit according to an embodiment of the presentinventive concept. When the sensing unit is disposed at the center ofthe display area DA, the virtual area VA is disposed in the display areaDA, and the sensing center line SCL and the display center area SCL areidentical to each other.

FIGS. 18A and 18B are views illustrating a virtual point displayed on adisplay area of a display unit in an electronic apparatus according toan embodiment of the present inventive concept. When an object Ob issensed to correspond to one of icons Ma, Mb, and Mc, a symbol SYM isdisplayed at the corresponding icon Ma, Mb, or MC on the display areaDA, as illustrated in FIG. 18A. The symbol SYM representing the sensedobject may be disposed at a center portion or a side portion of an areaof the corresponding icon Ma, Mb, and Mc. When the object is sensed tobe disposed at a portion which does not correspond to one of the iconsMa, Mb, and Mc, it is possible that a symbol SYM may be displayed at apoint outside the icons MA, Mb, or Mc, as illustrated in FIG. 18B. Thesymbol SYM may be a dot, a flash, a brink, or a sign. However, thepresent general inventive concept is not limited thereto. The symbol maybe a characteristic change of the corresponding one of the icons Ma, Mb,and MC while the other icons do not show any characteristic change.

FIGS. 19 and 20 are flowcharts illustrating a method of inputting a usercommand using a sensing unit in an electronic apparatus according to anembodiment of the present inventive concept.

Referring to FIG. 19, when a sensing unit, for example, a camera, isturned on, activated, or starts one of functions thereof in operation1910, an object exists in the photographed image in operation 1920. Avirtual area is extracted from the photographed image in operation 1930.An object is extracted from the virtual area in operation 1940. Thestatus of the object is analyzed and determined as a user input when theobject is disposed at a position corresponding to an icon displayed on adisplay area in operation 1950. A function corresponding to the icon isperformed in operation 1960.

Referring to FIG. 20, when a sensing unit, for example, a camera, isturned on, activated, or starts one of functions thereof in operation2010, an object is detected from a virtual area corresponding to adisplay area in operation 2020. A coordinate of the virtual areacorresponds to a coordinate of the display area. When a position of theobject with respect to the coordinate of the virtual area corresponds toa position of an icon with respect to the coordinate of the displayarea, it is determined that the presence, existence, or movement of theobject is a user input in operation 2030. A function of the icon isperformed in operation 2040.

FIG. 21 is a view illustrating a location of a sensing unit in anelectronic apparatus 2100 according to an embodiment of the presentinventive concept. The sensing unit may be one or more sensing elementsdisposed positions 2103 a, 2103 b, 2013 c, and 2103 d around a displayarea DA of a display panel 2102 on a housing 2101. When one or morecameras are included in the sensing unit, the cameras may be installedat positions 2104 and at least one of the positions 2103 a, 2103 b, 2013c, and 2103 d. It is possible that the sensing element and the cameramay be same to photograph an object as a photographing mode and to sensean object as a user input with respect to the display area DA in asensing mode. Accordingly, the camera can be usable in one of thephotographing mode and the sensing mode or also usable in both thephotographing mode and the sensing mode at the same time. In this case,an image corresponding to the photographed object has a centercorresponding to the sensing center line SCL, and an image correspondingto the sensed object in the virtual area VA may have a centercorresponding to the display center line SCL. When the object disposedat a position on the display center line DCL is photographed anddisplayed at a position different from the display center line DCL.However, when the object disposed at a position on the display centerline DCL is sensed and analyzed to be disposed at a position on thedisplay center line SCL according to coordinates of the display area DAand the virtual area VA.

FIGS. 22 and 23 are views illustrating an electronic apparatus having asensing unit installed thereon according to an embodiment of the presentgeneral inventive concept. FIG. 22 illustrates that a sensing unit isinstalled at a side of the display area DA such that a sensing centerline SCL is perpendicular to the display area DA and a boundary line BLis inclined to spaced apart from the display area according to adistance from the sensing center line SCL or the sensing unit. FIG. 23illustrates that a sensing unit is installed at a side of the displayarea DA such that a sensing center line SCL is not perpendicular to thedisplay area DA. The sensing unit may be installed such that a boundaryline BL can be substantially parallel to the display area DA. However,the present general inventive concept is not limited thereto. Theboundary line BL may be spaced apart from the display area DA by adistance so that the sending unit can sense a user finger or tooldisposed adjacent to the display area DA. The sensing unit may have asub-housing with a lens unit having the sensing center line SC, and thesub-housing may be installed in the sub-housing at an inclination anglewith a plane of the display area DA. A method of installing a sensingunit on a housing is well known, detail description thereof is omitted.It is possible that the boundary line BL of FIG. 23 may be inclined withrespect to the display area DA. In this case, an inclination angle IAbof the boundary line BL of FIG. 23 is smaller than an inclination angleIAa of the boundary line BL of FIG. 22. A hatched portion is a virtualarea of a sensing area disposed above the display area DA in a directionperpendicular to the display area DA.

The present general inventive concept can also be embodied ascomputer-readable codes on a computer-readable medium. Thecomputer-readable medium can include a computer-readable recordingmedium and a computer-readable transmission medium. Thecomputer-readable recording medium is any data storage device that canstore data as a program which can be thereafter read by a computersystem. Examples of the computer-readable recording medium include asemiconductor device, a read-only memory (ROM), a random-access memory(RAM), a flash memory, a semiconductor chip package, CD-ROMs, magnetictapes, floppy disks, and optical data storage devices. Thecomputer-readable recording medium can also be distributed over networkcoupled computer systems so that the computer-readable code is storedand executed in a distributed fashion. The computer-readabletransmission medium can transmit carrier waves or signals (e.g., wiredor wireless data transmission through the Internet). Also, functionalprograms, codes, and code segments to accomplish the present generalinventive concept can be easily construed by programmers skilled in theart to which the present general inventive concept pertains.

As described above and also illustrated in the drawings, a user mayinput a user command on a virtual area corresponding to a display areawithout touching the display area in an electronic apparatus accordingto an embodiment of the present inventive concept. Also, the virtualarea may be spaced apart from the display area, disposed parallel to thedisplay area, and disposed to overlap the display area in a verticaldirection from the display area in the electronic apparatus such that anobject can be sensed to correspond to the same point in the virtual areaand the display area in the vertical direction according to anembodiment of the present general inventive concept.

Moreover, as described above and also illustrated in the drawings, theelectronic apparatus may include a housing, a display unit installed inthe housing and having a display area to display an image or a menuicon, a sensing unit installed in the housing adjacent to the displayunit, and having a virtual area to sense an object which is disposedover the display area of the display unit, the virtual area defined by aboundary extended from the display area in a direction perpendicular tothe display area, and a controller to analyze status of the objectaccording to an original state of the object and a sensed state of theobject, and to determine the analyzed status of the object as a userinput.

The original state of the object and the sensed state of the object withrespect to the display area may be different from each other accordingto a difference between center lines of the sensing unit and the displayarea of the display unit.

The controller may adjust the sensed state of the object to the originalstate of the object with respect to the display area to determinewhether the status of the object corresponds to the user input to thedisplay area.

The controller may control the display unit to display a symbolrepresenting the status of the object with respect to the display area.

The symbol may be displayed on the menu icon or outside the menu icon inthe display area according to a location of the status of the objectwith respect to the display area.

The virtual area and the display area may overlap in the boundary in theperpendicular direction.

The menu icon may be disposed in the display area, and the object may bedisposed in the virtual area to correspond to the menu icon of thedisplay area.

The sensing unit may have a sensing area with a sensing center line andthe display area has a display center line which is spaced apart fromthe sensing center line by a distance.

The virtual area may be disposed in a side of the sensing area withrespect to the sensing center line, and the controller may extract thevirtual area from the sensing area.

When the object moves a first distance in a first direction within thevirtual area, the sensing unit may sense the object moving a seconddistance in a second distance since a sensing center line of the sensingunit is not a center line of the virtual area, and the controller mayanalyze the status of the object according to differences between thefirst and second distances and between the first and second distancesand determines the status of the object as the user input with respectto the display area.

When the object moves toward a first position of the display area, thesensing unit may sense the object moving toward a second position of thedisplay area in the virtual area, the second position being differentfrom the first position. The controller may adjust the second positionto the first position with respect to the display area.

The sensing unit may sense the object at a first position with respectto the virtual area, and the controller may convert the sensed firstposition into a second position with respect to the display area as thestatus of the object and determines the status of the object as the userinput.

The display area and the virtual area may have a same coordinate, thesensing unit may sense the object in the coordinate of the virtual area,and the controller may determine the status of the object as the userinput from the sensed object according to the coordinate of the displayarea.

The virtual area is disposed to overlap the display area in a verticaldirection from the display area, and the controller may determinewhether the status of the sensed object of the virtual area correspondsto the menu icon of the display area, and also determine the status ofthe sensed object as the user input of the menu icon to perform afunction of the menu icon.

The sensing unit may sense one or more sub-objects, and the controllermay determine one of the one or more sub-objects as the object.

The sensing unit may sense a user face and a user finger, and thecontroller may determine the user finger as the object.

The sensing unit may sense a user finger as the object, and thecontroller determines a distal end of the user finger as the status ofthe object.

The sensing unit may have a sensing center line to have an angle with adisplay center line of the display area, and the sensing center line ofthe camera may not be parallel to the display center line of the displayarea.

The sensing unit may be a camera having a photographing area tophotograph the object. The controller may extract from the photographingarea the virtual area including the object disposed over the displayarea, and may also determine from the photographed image that the objectis disposed over the menu icon in a direction perpendicular directionfrom the display area.

The display unit may be a touch panel usable as an output unit todisplay the image or the menu icon and an input unit to receive the userinput to control the image or the menu icon. The sensing unit mayinclude a lens unit with a sensing area in which the virtual area and anon-virtual area are included, and the lens unit includes a sensor tophotograph the virtual area.

The display unit may include a display panel having the display area,and the sensing unit is disposed at a bottom area of the display panelof the display unit.

Furthermore, as described above and also illustrated in the drawings,The electronic apparatus may include a housing, a display unit installedin the housing, and having a display area usable as an output unit todisplay an image or a menu icon and an input unit to receive a userinput to control a function of the electronic apparatus including theimage or the menu icon, a sensing unit installed in the housing adjacentto the display unit, and having a camera with a photographing areaincluding a virtual area to sense an object which is disposed over thedisplay area of the display unit, the virtual area defined by a boundaryextended from the display area in a direction perpendicular to thedisplay area, and a controller to analyze status of the object accordingto an original state of the object and a sensed state of the object, andto determine the analyzed status of the object as the user input.

Furthermore, as described above and also illustrated in the drawings,the electronic apparatus may perform a method including providing ahousing with a display unit having a display area to display an image ora menu icon, and a sensing unit installed adjacent to the display unitand having a virtual area to sense an object which is disposed over thedisplay area of the display unit, the virtual area defined by a boundaryextended from the display area in a direction perpendicular to thedisplay area, and analyzing status of the object according to anoriginal state of the object and a sensed state of the object, anddetermining the analyzed status of the object as a user input.

Furthermore, as described above and also illustrated in the drawings,the present general inventive concept may be implemented as acomputer-readable medium to contain computer-readable codes as a programto perform the method described above in the electronic apparatus.

Although a few exemplary embodiments of the present general inventiveconcept have been shown and described, it will be appreciated by thoseskilled in the art that changes may be made in these exemplaryembodiments without departing from the principles and spirit of thegeneral inventive concept, the scope of which is defined in the appendedclaims and their equivalents.

What is claimed is:
 1. A mobile device, comprising: a housing having afront side and a rear side; an input and output installed in thehousing, and comprising a panel on the front side of the housing toreceive a user input and to display a photographing image or a menu; asensing unit installed on the front side of the housing, and having afirst camera and a second camera respectively having a first virtualarea and a second virtual area which correspond to a display area of thepanel; and a controller installed in the housing, configured to performa photographing mode to control the first camera to photograph the firstvirtual area corresponding to the display area of the panel to generatea first photographing image, and to control the display unit to displaythe first photographing image in the display area of the panel, andconfigured to perform a sensing mode to control the second camera tophotograph the second virtual area including an object disposed to facethe display area of the panel to generate a second photographing imagerepresenting the object, to determine a state of the object of thesecond photographing image, and to perform a function of the mobiledevice according to the menu and the state of the object as an input tothe menu, wherein the second virtual area includes a plurality ofsub-objects, one of which has a higher priority than the other one, andthe controller is configured to determine a state of the one sub-objectwith the higher priority as the input to the menu.
 2. The mobile deviceof claim 1, further comprising: a memory to store information, whereinthe controller is configured to compare the information and the state ofthe object and to perform the function of the mobile device according tothe menu and the comparison.
 3. The mobile device of claim 1, whereinthe controller is configured to analyze the object of the secondphotographing image with respect to the display area of the panel tosense the state of the object, and to perform the function of the mobiledevice according to the menu and the analysis.
 4. The mobile device ofclaim 1, wherein the controller is configured to modify the object ofthe second photographing image with respect to the display area of thepanel to sense the state of the object, and to perform the function ofthe mobile device according to the menu and the modification.
 5. Themobile device of claim 1, wherein the second camera photographs theobject of the second virtual area when the menu is displayed in thedisplay area of the panel.
 6. The mobile device of claim 1, wherein thesecond camera generates the second photographing image corresponding tothe second virtual area, and the controller extracts the object from thesecond photographing image to sense the state of the object and performsthe function of the mobile device according to the menu and theextraction.
 7. The mobile device of claim 1, wherein the controller isconfigured to display an image corresponding to at least one of thestate of the object and the state of the one sub-object over the menu ofthe display area of the panel.
 8. The mobile device of claim 1, whereinthe controller is configured to control the input and output unit togenerate an output of the performed function of the mobile device, andthe output comprises an image output through the display area of thepanel, an audio output through an audio element, or a data outputthrough an interface to an external device.
 9. The mobile device ofclaim 1, further comprising: an interface configured to communicate withan external device to transmit or receive data to perform the functionof the mobile device, wherein the controller is configured to performthe function of the mobile device according to the menu of the input andoutput unit, the state of the object as the user input to the menu, andthe data of the external device.
 10. The mobile device of claim 1,wherein: the controller selects the one sub-object with the higherpriority as the object corresponding to the menu; and the controllerperforms the function according to at least one of the state of theobject and the state of the one sub-object with the higher priority. 11.A method of a mobile device, the method comprising: providing a housinghaving a front side and a rear side, an input and output installed inthe housing and comprising a panel on the front side of the housing toreceive a user input and to display a photographing image or a menu, anda sensing unit installed on the front side of the housing, and having afirst camera and a second camera each respectively having a firstvirtual area and a second virtual area which correspond to a displayarea of the panel of the input and output unit; using a controllerprocessor installed in the housing, configured to perform aphotographing mode to control the first camera to photograph the firstvirtual area corresponding to the display area of the panel to generatea first photographing image, and to control the display unit to displaythe first photographing image in the display area of the panel, andconfigured to perform a sensing mode to control the second camera tophotograph the second virtual area including an object disposed to facethe display area of the panel to generate a second photographing imagerepresenting the object, to determine a state of the object of thesecond photographing image, and to perform a function of the mobiledevice according to the menu and the state of the object as an input tothe menu, wherein the second virtual area includes a plurality ofsub-objects, one of which has a higher priority than the other one, andthe method further comprises determining a state of the one sub-objectwith the higher priority as the input to the menu.
 12. The method ofclaim 11, wherein: the mobile device further comprises a memory to storeinformation; and the method further comprises comparing the informationand the state of the object and to perform the function of the mobiledevice according to the menu and the comparison.
 13. The method of claim11, wherein the method further comprises analyzing the object of thesecond photographing image to sense the state of the object, andperforming the function of the mobile device according to the menu andthe analysis.
 14. The method of claim 11, wherein the method furthercomprises modifying the object of the second photographing image withrespect to the display area of the panel to sense the state of theobject, and performing the function of the mobile device according tothe menu and the modification.
 15. The method of claim 11, wherein thecontrolling the second camera comprises controlling the second camera tophotograph the object of the second virtual area during displaying themenu in the display area of the panel.
 16. The method of claim 11,further comprising: displaying an image corresponding to at least one ofthe state of the object and the state of the one sub-object over themenu of the display area of the panel.
 17. The method of claim 11,further comprising: controlling the input and output unit to generate anoutput of the performed function of the mobile device, wherein theoutput comprises an image output through the display area of the panel,an audio output through an audio element, or a data output through aninterface to an external device.
 18. The method of claim 11, furthercomprising: an interface configured to communicate with an externaldevice to transmit or receive data to perform a function of the mobiledevice, wherein the controller is configured to perform the function ofthe mobile device according to the menu, the state of the object as theuser input to the menu, and the data of the external device.
 19. Themethod of claim 11, wherein: the method further comprises selecting theone sub-object with the higher priority as the object corresponding tothe menu; and the method further comprises performing the functionaccording to at least one of the state of the object and the state ofthe one sub-object with the higher priority.
 20. A non-transitorycomputer-readable medium containing computer-readable codes as a programto perform the method of claim 11.